On the Construction of Initial Basis Function for Efficient Value Function Approximation

We address the issues of improving the feature generation methods for the value-function approximation and the state space approximation. We focus the improvement of feature generation methods on approaches based on the Bellman error. The original Bellman-error-based approaches construct the first basis function as an arbitrary nonzero vector. This kind of design results an inefficient generation of the basis functions. We propose a method to construct the first basis function that models the structure of the value-function. Our method improves the efficiency of existing feature generation algorithms and derives a more precise model for value-function approximation. We also propose to use the relevance vector machine to find a sparse state representation and project the original high-dimensional state space to the resulting low-dimensional state space. Our framework shows improved performance on existing benchmark problems, and is also effective on a car racing problem.